Skip to content

Fix Python 3.12+ fork warning in async Task SDK connection tests #56019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kaxil merged 1 commit into from
Sep 24, 2025
Merged

Fix Python 3.12+ fork warning in async Task SDK connection tests #56019

kaxil merged 1 commit into from
Sep 24, 2025

Conversation

@kaxil
Copy link
Member

@kaxil kaxil commented Sep 24, 2025 •
edited
Loading

We encountered an intermittent test failure on Python 3.12 only in the Task SDK: https://github.com/apache/airflow/actions/runs/17957802850/job/51079451184#step:9:696 and marked that test as XFAIL: 6d2aac6

DeprecationWarning: This process (pid=78) is multi-threaded, use of fork() may lead to deadlocks in the child.

Failed Test: `

task-sdk/tests/task_sdk/execution_time/test_supervisor.py::TestWatchedSubprocess::test_reading_from_pipes

Example failure: https://github.com/apache/airflow/actions/runs/17957802850/job/51079451184#step:9:696

Root Cause Analysis

The Threading Issue

Python 3.12 introduced a DeprecationWarning when os.fork() is called in a multi-threaded process.

Code Path Investigation

Through systematic analysis, we discovered:

  1. Source: asgiref.sync.sync_to_async creates persistent ThreadPoolExecutor threads
  2. Location: task-sdk/src/airflow/sdk/execution_time/context.py:217 
    conn = await sync_to_async(secrets_backend.get_connection)(conn_id)
  3. Flow: 
    test_async_get_connection_from_api
  └── _async_get_connection()
      └── ensure_secrets_backend_loaded()
          └── sync_to_async(secrets_backend.get_connection)  # Creates ThreadPoolExecutor
              └── ThreadPoolExecutor thread persists
                  └── Next test: test_reading_from_pipes
                      └── os.fork() triggers Python 3.12 warning

Testing Instructions for Reviewers:

# Test the specific problematic sequence
breeze run pytest task-sdk/tests/task_sdk/execution_time/test_context_cache.py::TestAsyncConnectionCache::test_async_get_connection_from_api task-sdk/tests/task_sdk/execution_time/test_supervisor.py::TestWatchedSubprocess::test_reading_from_pipes -v

# Test broader async test suite  
breeze run pytest task-sdk/tests/task_sdk/execution_time/ -k asyncio

Example run:

root@7bbad33290d7:/opt/airflow# pytest  task-sdk/tests/task_sdk/execution_time/test_context_cache.py::TestAsyncConnectionCache::test_async_get_connection_from_api   task-sdk/tests/task_sdk/execution_time/test_supervisor.py::TestWatchedSubprocess::test_reading_from_pipes
================================================================================================ test session starts =================================================================================================
platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.6.0 -- /usr/local/bin/python3
cachedir: .pytest_cache
rootdir: /opt/airflow/task-sdk
configfile: pyproject.toml
plugins: kgb-7.2, asyncio-1.2.0, cov-7.0.0, custom-exit-code-0.3.0, icdiff-0.9, instafail-0.5.0, mock-3.15.1, rerunfailures-16.0.1, timeouts-1.2.1, unordered-0.7.0, requests-mock-1.12.1, xdist-3.8.0, time-machine-2.19.0, anyio-4.11.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 2 items

task-sdk/tests/task_sdk/execution_time/test_context_cache.py::TestAsyncConnectionCache::test_async_get_connection_from_api PASSED                                                                              [ 50%]
task-sdk/tests/task_sdk/execution_time/test_supervisor.py::TestWatchedSubprocess::test_reading_from_pipes FAILED                                                                                               [100%]

====================================================================================================== FAILURES ======================================================================================================
___________________________________________________________________________________ TestWatchedSubprocess.test_reading_from_pipes ____________________________________________________________________________________
task-sdk/tests/task_sdk/execution_time/test_supervisor.py:274: in test_reading_from_pipes
    assert captured_logs == unordered(
E   assert [{'category': 'DeprecationWarning', 'event': 'This process (pid=3505) is multi-threaded, use of fork() may lead to dea...lsite', 'filename': '/opt/airflow/task-sdk/tests/task_sdk/execution_time/test_supervisor.py', 'level': 'warning', ...}] == [{'logger': 'task.stdout', 'event': "I'm a short message", 'level': 'info', 'timestamp': '2024-11-07T12:34:56.078901Z'... 249, 'logger': 'py.warnings', 'timestamp': datetime.datetime(2024, 11, 7, 12, 34, 56, 78901, tzinfo=Timezone('UTC'))}]
E     Extra items in the left sequence:
E     {'category': 'DeprecationWarning', 'event': 'This process (pid=3505) is multi-threaded, use of fork() may lead to dead...the child.', 'filename': '/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py', 'level': 'warning', ...}

I do not want to admit how much time I & @jedcunningham spent on debugging this frustratingly :)

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kaxil
Fix Python 3.12+ fork warning in async connection tests
006c457 
Resolves intermittent test failure where `asgiref.sync.sync_to_async` creates
ThreadPoolExecutors that persist between tests, causing Python 3.12+ to warn
about forking multi-threaded processes when supervisor.py calls `os.fork()`.

Added targeted cleanup of asgiref executors after async tests to ensure
clean thread environment for subsequent tests that use `fork()`.

Example failure: https://github.com/apache/airflow/actions/runs/17957802850/job/51079451184#step:9:696
@kaxil kaxil requested a review from jedcunningham September 24, 2025 00:59
@kaxil
Copy link
Member Author

kaxil commented Sep 24, 2025

Failures are due to docker rate limit:

ERROR providers/mongo/tests/unit/mongo/hooks/test_mongo.py::TestMongoHook::test_invalid_conn_type_srv - docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.48/images/create?tag=latest&fromImage=mongo: Internal Server Error ("toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit")

amoghrajesh
amoghrajesh approved these changes Sep 24, 2025
View reviewed changes
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a strange one! Was it introduced "now" due to some dependency bump or it was just a matter of fact till we eventually hit it?

@kaxil kaxil merged commit 8e6f03e into apache:main Sep 24, 2025
76 of 77 checks passed
@kaxil kaxil deleted the add-fix-async branch September 24, 2025 07:22
kaxil added a commit that referenced this pull request Sep 26, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (#56019)
5f83aed 
(cherry picked from commit 8e6f03e)
@kaxil kaxil added this to the Airflow 3.1.1 milestone Sep 26, 2025
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Sep 30, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
0384dca
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 1, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
93aaeca
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 2, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
0a634aa
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 3, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
aada30e
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 4, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
5e79d7c
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
8d57db4
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
07118a1
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
d1587d4
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
09d6303
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
66a468f
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
724b8d1
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
2ea0b00
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
b303b03
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
24f899f
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
3fe390d
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
8342ab7
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
@kaxil
Fix Python 3.12+ fork warning in async connection tests (apache#56019)
a605492
@kaxil kaxil mentioned this pull request Oct 21, 2025
Merged
@potiuk potiuk mentioned this pull request Nov 8, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@amoghrajesh amoghrajesh amoghrajesh approved these changes

@jedcunningham jedcunningham jedcunningham approved these changes

@ashb ashb Awaiting requested review from ashb ashb is a code owner

Assignees

No one assigned

Labels

area:task-sdk

Projects

None yet

Milestone

Airflow 3.1.1

Development

Successfully merging this pull request may close these issues.

3 participants

@kaxil @amoghrajesh @jedcunningham